Theo_Nazon_SDI701_Cover_Type¶
// TOOL USED - SCI-KIT LEARN //¶
Competition objective¶
Predict the forest cover type (the predominant kind of tree cover) from strictly cartographic variables (as opposed to remotely sensed data).
Data provided¶
Raw form (not scaled) and contains binary columns of data for qualitative independent variables such as wilderness areas and soil type. We are dealing with a classification problem, containing labeled data (from 1 to 7 corresponding to the different Cover Type). Additionnaly, trainset includes ~500k samples, and data is numeric (we are not dealing with text data for which Naive Bayes could have been favored choice). Initial thougths leads to consider Ensemble Classifiers (RandomForest, XGBoost, Extratrees). Our goal here will be to compare these classifiers and tune their parameters to reach and optimal solution
Support¶
Given algorithms considered for implemententation, choice naturally turned to sci-kit library. Indeed, ExtraTree is not implemented in pyspark ML Libraries
===================================================================================================================
Workflow¶
- Data Loading
- Data Preprocessing
- Objective : Prepare data for visualization and training part.
- Comment : Data is already very well processed (binary columns etc.) though it is not normalized. We will leave it as it is as we consider using decision tree based classifier which do not require normalization
- Data visualization
- Objective : Identify most important variables for possible feature engineering
- Comment : Data already contains around XX features. Data visualization will help us assess if some variable are particularly important (as defined by the various classification models – see Jupyter Notebook for further details), and/or if we can drop some non-important feature that could lead to model over-fititing
- Feature engineering
- Objective : Increase model performance D) Training modeling Comment : We will first try out a Random Forest Classifier on the train set. Reasons being : I) it is the one I understand the most clearly and I’d rather use something I understand first, ii) it also measures feature importance which will be used in part ii) and iii).
===================================================================================================================
Key take-away¶
- Ensemble Classifier do a great job at classifying this data, reaching 94% and ~96% using Random Forest and ExtraTreeClassifier
- Hyperparameter tuning helps improve scores by [2% - 5%]
- Engineered features improves marginally models performances:
- Geological / Climatic Zone have no to negative impact on performance, certainly explained by the very low importance level of Soil Type for predicting the cover type
- Filling up Hillshade_3pm has no impact, which was anticipated considering the very small amount of negative or zero values
- Running time is faster in ExtraClassifier compared to Random Forest
- Both algorithm are very easy to use :
- Manage not scaled data / categorical / binary / continuous data
- Require few parameter tuning work
- Prevent overfitting through the use of bagging / random features selection / random sampling
===================================================================================================================
Import¶
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib as mpl
from scipy.interpolate import BSpline
import numpy as np
import math as m
import random
import scipy.stats as stats
from scipy.stats import norm
import matplotlib.lines as mlines
import statistics as stat
from sklearn import ensemble
from pandas.plotting import scatter_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import KFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import f1_score
from sklearn.ensemble import ExtraTreesClassifier
import statsmodels as sm
%matplotlib inline
plt.style.use('seaborn')
sns.set_style('darkgrid')
# Ignore message error arising on sns.pairplot
np.seterr(divide='ignore', invalid='ignore')
0. DATA LOADING¶
Setting seed parameters¶
train_test_seed = 0
Loading train and test datasets provided by Kaggle¶
train = pd.read_csv("/home/theo/Dropbox/01. MASTER TELECOM PARIS/01. SD701 - Exploration de grands volumes de données/Final_Test/train-set.csv")
test = pd.read_csv("/home/theo/Dropbox/01. MASTER TELECOM PARIS/01. SD701 - Exploration de grands volumes de données/Final_Test/test-set.csv")
Loading additional data from US Forest Service on climatic zone and geological zonetrain¶
geological_zone = pd.read_csv("/home/theo/Dropbox/01. MASTER TELECOM PARIS/01. SD701 - Exploration de grands volumes de données/Final_Test/geological_zone.csv")
climatic_zone = pd.read_csv("/home/theo/Dropbox/01. MASTER TELECOM PARIS/01. SD701 - Exploration de grands volumes de données/Final_Test/climatic_zone.csv")
usfs_corr_tab = pd.read_csv("/home/theo/Dropbox/01. MASTER TELECOM PARIS/01. SD701 - Exploration de grands volumes de données/Final_Test/usfs_elu_code.csv", sep=";")
1. DATA PREPROCESSING¶
Creating a DataFrame containing the cover type indices and the corresponding labels for vizualisation purpose¶
cover_type = train["Cover_Type"].unique()
cover_type.sort(axis=0)
cover_type = cover_type.tolist()
cover_colors = ["lightseagreen", "sandybrown", "lightgreen", "aquamarine", "darkgoldenrod", "lightsteelblue", "tomato"]
cover_labels = ["Spruce/Fir", "Lodgepole Pine", "Ponderosa Pine", "Cottonwood/Willow", "Aspen", "Douglas-fir", "Krummholz"]
df_cover_type = pd.DataFrame({"cover_indice" : cover_type, "cover_label" : cover_labels, "cover_color" : cover_colors})
Adding a column containing the labels of the cover type in the train DataFrame¶
train['Cover_Type_lab'] = train['Cover_Type'].map(df_cover_type.set_index('cover_indice')['cover_label'])
Saving train/test ID column in a Series before dropping the column from the train DataFrame (not a feature not be considered)¶
ID_train = train["Id"]
train.drop('Id', axis=1, inplace=True)
ID_test = test["Id"]
test.drop("Id", axis=1, inplace=True)
Creating the feature matrix X and the observation vector Y from the train dataset¶
X = train.drop(["Cover_Type", "Cover_Type_lab"], axis=1)
Y = train["Cover_Type"]
2. DATA VISUALIZATION¶
2.1 Approach¶
We are dealing with a classification problem. A first step prior to (if relevant) proceed with feature engineering is to analyse the most important variable and the correlation between them
To do so, we will train a basic Random Forest Classifier on the train data, and analyse the most important features
initial_rf_clf = RandomForestClassifier(n_estimators=200, max_depth=40, random_state=0, n_jobs=-1)
initial_rf_clf.fit(X, Y)
The following command will extract the importance of each feature after the Random Forest classifier has been trained on the data
"Feature importance is calculated as the decrease in node impurity weighted by the probability of reaching that node. The node probability can be calculated by the number of samples that reach the node, divided by the total number of samples. The higher the value the more important the feature."
features_importance = initial_rf_clf.feature_importances_
2.2 Plotting the Top 10 features by importance¶
Creating a DataFrame containing the features, their importance, ordered by decreasing order¶
# Removing Cover_Type indices and labels from the train columns to get the features labels
features_labels = train.columns[:-2]
df_features = pd.DataFrame({"feature" : features_labels, "importance" : features_importance})
df_features.sort_values("importance", ascending=True, inplace=True)
Creating a DataFrame containing the Top 10 features by order of importance and plotting them¶
df_top10_features = df_features.tail(10)
y_pos = np.arange(df_top10_features.shape[0])
plt.barh(y_pos, df_top10_features["importance"], align='center', alpha=0.5)
plt.yticks(y_pos, df_top10_features["feature"])
plt.xlabel('Importance level')
plt.title('Top 10 Feature by importance (via Random Forest)')
plt.show()
===================================================================================================================
Key take-away¶
- Using a RandomForestClassifier to get the feature importance in the classification process, | Elevation | is clearly a determining factor in the process of determining the cover type. Next step would be to plot the covariance plot of the most important features to identify if there are any correlation / relationship that could push for relevant feature engineering
- The 10 continuous features are topping the ranking. Reversely, soil_type does not position itself as an important feature
===================================================================================================================
2.3 Plotting the correlation matrix of the Top 10 features by importance¶
columns_to_select = df_top10_features["feature"].append(pd.Series("Cover_Type_lab"))
train_top10 = train[columns_to_select]
matrix = sns.pairplot(train_top10, size=5, hue="Cover_Type_lab", diag_kind="kde", diag_kws=dict(shade=True))
plt.tick_params(labelsize=10)
plt.show()
